{
  "nbformat": 4,
  "nbformat_minor": 0,
  "metadata": {
    "colab": {
      "provenance": [],
      "gpuType": "T4",
      "toc_visible": true
    },
    "kernelspec": {
      "name": "python3",
      "display_name": "Python 3"
    },
    "language_info": {
      "name": "python"
    },
    "accelerator": "GPU"
  },
  "cells": [
    {
      "cell_type": "markdown",
      "source": [
        "# AutoGPTQ\n",
        "\n",
        "AutoGPTQ is a user-friendly quantization package for Large Language Models (LLMs) based on the GPTQ algorithm. The purpose of this package is to allow users to easily quantize and optimize their language models for inference, making them faster and more efficient while preserving acceptable model quality."
      ],
      "metadata": {
        "id": "ihaZyg2aYQpH"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "| Code Credits | Link |\n",
        "| ----------- | ---- |\n",
        "| 🎉 Repository | [![GitHub Repository](https://img.shields.io/github/stars/PanQiWei/AutoGPTQ?style=social)](https://github.com/PanQiWei/AutoGPTQ) |\n",
        "| 🔥 Discover More Colab Notebooks | [![GitHub Repository](https://img.shields.io/badge/GitHub-Repository-black?style=flat-square&logo=github)](https://github.com/R3gm/InsightSolver-Colab/) |\n"
      ],
      "metadata": {
        "id": "OBnzlAqYZM6_"
      }
    },
    {
      "cell_type": "code",
      "execution_count": null,
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "AppOVzGOJX0m",
        "outputId": "da6d84a6-8e5e-4847-fb3e-6094c31e36bc"
      },
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Requirement already satisfied: auto-gptq==0.2.2 in /usr/local/lib/python3.10/dist-packages (0.2.2)\n",
            "Requirement already satisfied: accelerate>=0.19.0 in /usr/local/lib/python3.10/dist-packages (from auto-gptq==0.2.2) (0.21.0)\n",
            "Requirement already satisfied: datasets in /usr/local/lib/python3.10/dist-packages (from auto-gptq==0.2.2) (2.13.1)\n",
            "Requirement already satisfied: numpy in /usr/local/lib/python3.10/dist-packages (from auto-gptq==0.2.2) (1.22.4)\n",
            "Requirement already satisfied: rouge in /usr/local/lib/python3.10/dist-packages (from auto-gptq==0.2.2) (1.0.1)\n",
            "Requirement already satisfied: torch>=1.13.0 in /usr/local/lib/python3.10/dist-packages (from auto-gptq==0.2.2) (2.0.1+cu118)\n",
            "Requirement already satisfied: safetensors in /usr/local/lib/python3.10/dist-packages (from auto-gptq==0.2.2) (0.3.1)\n",
            "Requirement already satisfied: transformers>=4.26.1 in /usr/local/lib/python3.10/dist-packages (from auto-gptq==0.2.2) (4.30.2)\n",
            "Requirement already satisfied: packaging>=20.0 in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.19.0->auto-gptq==0.2.2) (23.1)\n",
            "Requirement already satisfied: psutil in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.19.0->auto-gptq==0.2.2) (5.9.5)\n",
            "Requirement already satisfied: pyyaml in /usr/local/lib/python3.10/dist-packages (from accelerate>=0.19.0->auto-gptq==0.2.2) (6.0)\n",
            "Requirement already satisfied: filelock in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq==0.2.2) (3.12.2)\n",
            "Requirement already satisfied: typing-extensions in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq==0.2.2) (4.7.1)\n",
            "Requirement already satisfied: sympy in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq==0.2.2) (1.11.1)\n",
            "Requirement already satisfied: networkx in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq==0.2.2) (3.1)\n",
            "Requirement already satisfied: jinja2 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq==0.2.2) (3.1.2)\n",
            "Requirement already satisfied: triton==2.0.0 in /usr/local/lib/python3.10/dist-packages (from torch>=1.13.0->auto-gptq==0.2.2) (2.0.0)\n",
            "Requirement already satisfied: cmake in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.13.0->auto-gptq==0.2.2) (3.25.2)\n",
            "Requirement already satisfied: lit in /usr/local/lib/python3.10/dist-packages (from triton==2.0.0->torch>=1.13.0->auto-gptq==0.2.2) (16.0.6)\n",
            "Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.26.1->auto-gptq==0.2.2) (0.16.4)\n",
            "Requirement already satisfied: regex!=2019.12.17 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.26.1->auto-gptq==0.2.2) (2022.10.31)\n",
            "Requirement already satisfied: requests in /usr/local/lib/python3.10/dist-packages (from transformers>=4.26.1->auto-gptq==0.2.2) (2.27.1)\n",
            "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.26.1->auto-gptq==0.2.2) (0.13.3)\n",
            "Requirement already satisfied: tqdm>=4.27 in /usr/local/lib/python3.10/dist-packages (from transformers>=4.26.1->auto-gptq==0.2.2) (4.65.0)\n",
            "Requirement already satisfied: pyarrow>=8.0.0 in /usr/local/lib/python3.10/dist-packages (from datasets->auto-gptq==0.2.2) (9.0.0)\n",
            "Requirement already satisfied: dill<0.3.7,>=0.3.0 in /usr/local/lib/python3.10/dist-packages (from datasets->auto-gptq==0.2.2) (0.3.6)\n",
            "Requirement already satisfied: pandas in /usr/local/lib/python3.10/dist-packages (from datasets->auto-gptq==0.2.2) (1.5.3)\n",
            "Requirement already satisfied: xxhash in /usr/local/lib/python3.10/dist-packages (from datasets->auto-gptq==0.2.2) (3.2.0)\n",
            "Requirement already satisfied: multiprocess in /usr/local/lib/python3.10/dist-packages (from datasets->auto-gptq==0.2.2) (0.70.14)\n",
            "Requirement already satisfied: fsspec[http]>=2021.11.1 in /usr/local/lib/python3.10/dist-packages (from datasets->auto-gptq==0.2.2) (2023.6.0)\n",
            "Requirement already satisfied: aiohttp in /usr/local/lib/python3.10/dist-packages (from datasets->auto-gptq==0.2.2) (3.8.4)\n",
            "Requirement already satisfied: six in /usr/local/lib/python3.10/dist-packages (from rouge->auto-gptq==0.2.2) (1.16.0)\n",
            "Requirement already satisfied: attrs>=17.3.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq==0.2.2) (23.1.0)\n",
            "Requirement already satisfied: charset-normalizer<4.0,>=2.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq==0.2.2) (2.0.12)\n",
            "Requirement already satisfied: multidict<7.0,>=4.5 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq==0.2.2) (6.0.4)\n",
            "Requirement already satisfied: async-timeout<5.0,>=4.0.0a3 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq==0.2.2) (4.0.2)\n",
            "Requirement already satisfied: yarl<2.0,>=1.0 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq==0.2.2) (1.9.2)\n",
            "Requirement already satisfied: frozenlist>=1.1.1 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq==0.2.2) (1.3.3)\n",
            "Requirement already satisfied: aiosignal>=1.1.2 in /usr/local/lib/python3.10/dist-packages (from aiohttp->datasets->auto-gptq==0.2.2) (1.3.1)\n",
            "Requirement already satisfied: urllib3<1.27,>=1.21.1 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.26.1->auto-gptq==0.2.2) (1.26.16)\n",
            "Requirement already satisfied: certifi>=2017.4.17 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.26.1->auto-gptq==0.2.2) (2023.5.7)\n",
            "Requirement already satisfied: idna<4,>=2.5 in /usr/local/lib/python3.10/dist-packages (from requests->transformers>=4.26.1->auto-gptq==0.2.2) (3.4)\n",
            "Requirement already satisfied: MarkupSafe>=2.0 in /usr/local/lib/python3.10/dist-packages (from jinja2->torch>=1.13.0->auto-gptq==0.2.2) (2.1.3)\n",
            "Requirement already satisfied: python-dateutil>=2.8.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->auto-gptq==0.2.2) (2.8.2)\n",
            "Requirement already satisfied: pytz>=2020.1 in /usr/local/lib/python3.10/dist-packages (from pandas->datasets->auto-gptq==0.2.2) (2022.7.1)\n",
            "Requirement already satisfied: mpmath>=0.19 in /usr/local/lib/python3.10/dist-packages (from sympy->torch>=1.13.0->auto-gptq==0.2.2) (1.3.0)\n"
          ]
        }
      ],
      "source": [
        "pip install auto-gptq==0.2.2"
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Select the model"
      ],
      "metadata": {
        "id": "R11KqY7lW0yv"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "\n",
        "First, we need to specify the model to use. In Colab with T4 GPU, we can run models of up to 20B of parameters with all optimizations, but this may degrade the quality of the model's inference."
      ],
      "metadata": {
        "id": "CzJkCoRRbICP"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "In this case, we will use a [WizardCoder](https://huggingface.co/WizardLM/WizardCoder-15B-V1.0), which is a model optimized for generating code in different programming languages.\n",
        "\n",
        "\n",
        "![asd](https://raw.githubusercontent.com/nlpxucan/WizardLM/main/WizardCoder/imgs/pass1.png)\n",
        "\n",
        "\n",
        "\n",
        "\n"
      ],
      "metadata": {
        "id": "hHAz1yGZb4lq"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Model quantization"
      ],
      "metadata": {
        "id": "PdmNwEqjRcPY"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "The model needs more VRAM than we have available, so we have to quantize it. One alternative for this procedure is to use [GPTQ-for-LLaMa](https://github.com/qwopqwop200/GPTQ-for-LLaMa) or use the method provided by auto_gptq:\n"
      ],
      "metadata": {
        "id": "aEcBfs9sRbE_"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "### Demo quantization for facebook/opt-125m to 4bits ###\n",
        "from transformers import AutoTokenizer, TextGenerationPipeline\n",
        "from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\n",
        "\n",
        "pretrained_model_dir = \"facebook/opt-125m\"\n",
        "quantized_model_dir = \"opt-125m-4bit\"\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(pretrained_model_dir, use_fast=True)\n",
        "\n",
        "examples = [\n",
        "    tokenizer(\n",
        "        \"auto-gptq is an easy-to-use model quantization library with user-friendly apis, based on GPTQ algorithm.\"\n",
        "    )\n",
        "]\n",
        "\n",
        "quantize_config = BaseQuantizeConfig(\n",
        "    bits=4,  # quantize model to 4-bit\n",
        "    group_size=128,  # it is recommended to set the value to 128\n",
        "    desc_act=False,  # set to False can significantly speed up inference but the perplexity may slightly bad\n",
        ")\n",
        "\n",
        "# load un-quantized model, by default, the model will always be loaded into CPU memory\n",
        "model = AutoGPTQForCausalLM.from_pretrained(pretrained_model_dir, quantize_config)\n",
        "\n",
        "# quantize model, the examples should be list of dict whose keys can only be \"input_ids\" and \"attention_mask\"\n",
        "model.quantize(examples)\n",
        "\n",
        "# save quantized model\n",
        "model.save_quantized(quantized_model_dir)\n",
        "\n",
        "# save quantized model using safetensors\n",
        "model.save_quantized(quantized_model_dir, use_safetensors=True)"
      ],
      "metadata": {
        "id": "6t_9O4T5QtTy"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "However, we still need more VRAM to quantize the wizardcoder with auto_gptq."
      ],
      "metadata": {
        "id": "ODmd_8tgSKUm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "#  Quantized Models from the Hugging Face Community"
      ],
      "metadata": {
        "id": "GSeedwAFSay9"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Fortunately, the Hugging Face community provides quantized models in `safetensor format`. This allows us to use the model on the T4 GPU efficiently, effectively and more safely.\n",
        "\n",
        "There are various variations, but the ones that interest us are those based on the GPTQ algorithm.\n",
        "\n",
        "We can see the different variations that WizardCoder has [here](https://huggingface.co/models?search=wizardcoder).\n",
        "\n",
        "\n",
        "\n",
        "In this case, we will use the model called [WizardCoder-Guanaco-15B-V1.1-GPTQ](https://huggingface.co/TheBloke/WizardCoder-Guanaco-15B-V1.1-GPTQ), which is a finetuned model using the dataset from [openassistant-guanaco](https://huggingface.co/datasets/timdettmers/openassistant-guanaco)."
      ],
      "metadata": {
        "id": "4QwfjGFYRM4g"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "![img.png]()"
      ],
      "metadata": {
        "id": "aZ0haSO07wxp"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model_name_or_path = \"TheBloke/WizardCoder-Guanaco-15B-V1.1-GPTQ\"\n",
        "model_basename = \"gptq_model-4bit-128g\""
      ],
      "metadata": {
        "id": "oI-kXwg5bHF-"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "First, we load the model."
      ],
      "metadata": {
        "id": "8Td05XSuiWdI"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import AutoTokenizer, pipeline, logging\n",
        "from auto_gptq import AutoGPTQForCausalLM, BaseQuantizeConfig\n",
        "\n",
        "# AutoGPTQ supports using triton to speedup inference, but it currently only supports Linux | pip install auto-gptq[triton]\n",
        "use_triton = False\n",
        "\n",
        "tokenizer = AutoTokenizer.from_pretrained(model_name_or_path, use_fast=True)\n",
        "\n",
        "model = AutoGPTQForCausalLM.from_quantized(model_name_or_path,\n",
        "        model_basename=model_basename,\n",
        "        #revision=\"gptq-4bit-32g-actorder_True\", # from a specific branch, use the revision parameter, else commet this line\n",
        "        use_safetensors=True,\n",
        "        trust_remote_code=False,\n",
        "        device=\"cuda:0\",\n",
        "        use_triton=use_triton,\n",
        "        quantize_config=None\n",
        "        )"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "eekJ9-R5NQJy",
        "outputId": "0582133b-fe8c-481f-c055-2720ffd6bdef"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stderr",
          "text": [
            "WARNING:accelerate.utils.modeling:The safetensors archive passed at /root/.cache/huggingface/hub/models--TheBloke--WizardCoder-Guanaco-15B-V1.1-GPTQ/snapshots/29f7b2b409b85f3df9efa3e54087779d02f2dd32/gptq_model-4bit-128g.safetensors does not contain metadata. Make sure to save your model with the `save_pretrained` method. Defaulting to 'pt' metadata.\n",
            "WARNING:auto_gptq.modeling._base:GPTBigCodeGPTQForCausalLM hasn't fused attention module yet, will skip inject fused attention.\n",
            "WARNING:auto_gptq.modeling._base:GPTBigCodeGPTQForCausalLM hasn't fused mlp module yet, will skip inject fused mlp.\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "We can obtain information about the model's configuration with"
      ],
      "metadata": {
        "id": "EK7i2mUoT5Bg"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model.config"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "InhMrUKnT88S",
        "outputId": "48c06827-eb7a-472a-8151-420d5b2afe94"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "GPTBigCodeConfig {\n",
              "  \"_name_or_path\": \"TheBloke/WizardCoder-Guanaco-15B-V1.1-GPTQ\",\n",
              "  \"activation_function\": \"gelu\",\n",
              "  \"architectures\": [\n",
              "    \"GPTBigCodeForCausalLM\"\n",
              "  ],\n",
              "  \"attention_softmax_in_fp32\": true,\n",
              "  \"attn_pdrop\": 0.1,\n",
              "  \"bos_token_id\": 0,\n",
              "  \"embd_pdrop\": 0.1,\n",
              "  \"eos_token_id\": 0,\n",
              "  \"inference_runner\": 0,\n",
              "  \"initializer_range\": 0.02,\n",
              "  \"layer_norm_epsilon\": 1e-05,\n",
              "  \"max_batch_size\": null,\n",
              "  \"max_sequence_length\": null,\n",
              "  \"model_type\": \"gpt_bigcode\",\n",
              "  \"multi_query\": true,\n",
              "  \"n_embd\": 6144,\n",
              "  \"n_head\": 48,\n",
              "  \"n_inner\": 24576,\n",
              "  \"n_layer\": 40,\n",
              "  \"n_positions\": 8192,\n",
              "  \"pad_key_length\": true,\n",
              "  \"pre_allocate_kv_cache\": false,\n",
              "  \"resid_pdrop\": 0.1,\n",
              "  \"scale_attention_softmax_in_fp32\": true,\n",
              "  \"scale_attn_weights\": true,\n",
              "  \"summary_activation\": null,\n",
              "  \"summary_first_dropout\": 0.1,\n",
              "  \"summary_proj_to_labels\": true,\n",
              "  \"summary_type\": \"cls_index\",\n",
              "  \"summary_use_proj\": true,\n",
              "  \"torch_dtype\": \"float16\",\n",
              "  \"transformers_version\": \"4.30.2\",\n",
              "  \"use_cache\": true,\n",
              "  \"validate_runner_input\": true,\n",
              "  \"vocab_size\": 49153\n",
              "}"
            ]
          },
          "metadata": {},
          "execution_count": 5
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "And the quantization configuration that was used."
      ],
      "metadata": {
        "id": "n6K_nhTiUCOv"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "model.quantize_config"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "f2bhhlhJUBxM",
        "outputId": "c6b0ec58-1877-4ec0-b592-dcafce8af88c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "execute_result",
          "data": {
            "text/plain": [
              "BaseQuantizeConfig(bits=4, group_size=128, damp_percent=0.01, desc_act=False, sym=True, true_sequential=True, model_name_or_path='TheBloke/WizardCoder-Guanaco-15B-V1.1-GPTQ', model_file_base_name='gptq_model-4bit-128g')"
            ]
          },
          "metadata": {},
          "execution_count": 6
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Inference"
      ],
      "metadata": {
        "id": "max1jwxvCSbm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "We will use this prompt."
      ],
      "metadata": {
        "id": "qLEEOufGVlID"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "prompt = \"Write a linear regresion in python\"\n",
        "prompt_template=f'''Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
        "\n",
        "### Instruction: {prompt}\n",
        "\n",
        "### Response:\n",
        "'''"
      ],
      "metadata": {
        "id": "-NzVIlMCVoVD"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Inference with transformers pipeline"
      ],
      "metadata": {
        "id": "j-jkIjITVDW6"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "logging.set_verbosity(logging.CRITICAL)\n",
        "\n",
        "pipe = pipeline(\n",
        "    task=\"text-generation\",\n",
        "    model=model,\n",
        "    tokenizer=tokenizer,\n",
        "    max_new_tokens=256,\n",
        "    temperature=0.7,\n",
        "    top_p=0.95,\n",
        "    repetition_penalty=1.15\n",
        ")\n",
        "\n",
        "print(pipe(prompt_template)[0]['generated_text'])\n"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "iIXvwGxbRn6F",
        "outputId": "744117b3-d1f2-4aee-89ac-81f58879974c"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
            "\n",
            "### Instruction: Write a linear regresion in python\n",
            "\n",
            "### Response:\n",
            "A Linear Regression model in Python can be implemented using scikit-learn library's `LinearRegression` class, which provides methods for fitting a regression line to data and predicting target values. Here are some steps you would follow when implementing this:\n",
            "\n",
            "1. Import necessary libraries like numpy and sklearn.linear_model\n",
            "    ```python\n",
            "    import numpy as np \n",
            "    from sklearn.linear_model import LinearRegression\n",
            "    ```\n",
            "    \n",
            "2. Generate random input data (X) with 100 samples and one independent variable (x). The relationship between X and y will be defined later on during training process.\n",
            "\n",
            "    ```python\n",
            "    x = np.random.rand(100,)  \n",
            "    noise = np.random.normal(loc=0, scale=5, size=(100,)) # add normal noise\n",
            "    y = 3*x + 4 + noise   # create a noisy sine wave signal\n",
            "    print(\"Shape of our dataset:\", x.shape, y.shape)\n",
            "    ```\n",
            "    \n",
            "    This code generates 100 random numbers ranging from [0,1] representing our independent variables 'x'. It then adds normally distributed noise to each point within range [-5,+5]. Finally it creates a noisy version of\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Inference with model.generate"
      ],
      "metadata": {
        "id": "3L4n6u7oCLxO"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "input_ids = tokenizer(prompt_template, return_tensors='pt').input_ids.cuda()\n",
        "\n",
        "output = model.generate(\n",
        "    inputs=input_ids,\n",
        "    max_new_tokens=256,\n",
        "    temperature=0.7,\n",
        "    top_p=0.95,\n",
        "    repetition_penalty=1.15\n",
        "    )\n",
        "\n",
        "print(tokenizer.decode(output[0]))"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "EWOF4VoDOqmw",
        "outputId": "2a0874f9-9f36-49bd-a11f-b09dca6d9088"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "Below is an instruction that describes a task. Write a response that appropriately completes the request.\n",
            "\n",
            "### Instruction: Write a linear regresion in python\n",
            "\n",
            "### Response:\n",
            "A Linear Regression model in Python can be implemented using scikit-learn library's `LinearRegression` class, which provides methods for fitting a regression line to data and predicting target values. Here are some steps you would follow when implementing this:\n",
            "\n",
            "1. Import necessary libraries like numpy and sklearn.linear_model\n",
            "    ```python\n",
            "    import numpy as np \n",
            "    from sklearn.linear_model import LinearRegression\n",
            "    ```\n",
            "    \n",
            "2. Generate random input data (X) with 100 samples and one independent variable (x). The relationship between X and y will be defined later on during training process.\n",
            "\n",
            "    ```python\n",
            "    x = np.random.rand(100,)  \n",
            "    noise = np.random.normal(loc=0, scale=5, size=(100,)) # add normal noise\n",
            "    y = 3*x + 4 + noise   # create a noisy sine wave signal\n",
            "    print(\"Shape of our dataset:\", x.shape, y.shape)\n",
            "    ```\n",
            "    \n",
            "    This code generates 100 random numbers ranging from [0,1] representing our independent variables 'x'. It then adds normally distributed noise to each point within range [-5,+5]. Finally it creates a noisy version of\n"
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "# Real-time output (stream tokens)"
      ],
      "metadata": {
        "id": "nQXhWYtNZM48"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "Real-time output allows continuous display of generated text as it is processed."
      ],
      "metadata": {
        "id": "HilKEtSGgPjm"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Basic token streaming\n",
        "\n"
      ],
      "metadata": {
        "id": "RVMq1jKOB5vg"
      }
    },
    {
      "cell_type": "markdown",
      "source": [
        "You can appreciate an example of how it is implemented in Gradio [here](https://huggingface.co/spaces/joaogante/transformers_streaming)."
      ],
      "metadata": {
        "id": "syyqQWYQYY7j"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import sys\n",
        "import time\n",
        "\n",
        "def real_time_print(string):\n",
        "    for char in string:\n",
        "        sys.stdout.write(char)\n",
        "        sys.stdout.flush()\n",
        "        time.sleep(0.1)  # adjust the sleep time to control the printing speed"
      ],
      "metadata": {
        "id": "GkiPYZc_2Z4V"
      },
      "execution_count": null,
      "outputs": []
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import AutoTokenizer, AutoModelForSeq2SeqLM, TextIteratorStreamer\n",
        "from threading import Thread\n",
        "\n",
        "model_inputs = tokenizer([prompt_template], return_tensors=\"pt\").to('cuda')\n",
        "\n",
        "# Adds timeout to the streamer to handle exceptions in the generation thread.\n",
        "streamer = TextIteratorStreamer(tokenizer, timeout=10., skip_prompt=True, skip_special_tokens=True)\n",
        "generate_kwargs = dict(\n",
        "    model_inputs,\n",
        "    streamer=streamer,\n",
        "    max_new_tokens=2000,\n",
        "    do_sample=True,\n",
        "    top_p=0.9,\n",
        "    temperature=float(0.7),\n",
        "    top_k=40\n",
        ")\n",
        "t = Thread(target=model.generate, kwargs=generate_kwargs)\n",
        "t.start()\n",
        "\n",
        "# Pull the generated text from the streamer, and update the model output.\n",
        "model_output = \"\"\n",
        "for new_text in streamer:\n",
        "    model_output += new_text\n",
        "    real_time_print(new_text)"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "7hI_HA200Ugr",
        "outputId": "bec372a9-1b12-43dc-985d-f8d6f77efb8e"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\n",
            "A linear regression in Python is a type of statistical analysis that uses a linear model to relate dependent variables to independent variables. Here's a simple example of how to do it using the statsmodels library:\n",
            "\n",
            "```python\n",
            "import numpy as np\n",
            "import pandas as pd\n",
            "from statsmodels.regression.linear_model import OLS\n",
            "\n",
            "# Create a dataset\n",
            "data = {'X1': [1, 2, 3, 4, 5], 'X2': [2, 3, 4, 5, 6], 'Y': [1, 2, 3, 4, 5]}\n",
            "df = pd.DataFrame(data)\n",
            "\n",
            "# Create a linear regression model\n",
            "X = df[['X1', 'X2']]\n",
            "y = df['Y']\n",
            "model = OLS(y, X)\n",
            "\n",
            "# Fit the model\n",
            "results = model.fit()\n",
            "\n",
            "# Print the coefficients\n",
            "print(results.params)\n",
            "```\n",
            "\n",
            "In this code, the `OLS` class is used to create a linear regression model. The `fit` method is then used to fit the model, which returns a results object with various statistical measures. The coefficients of the model are accessed through the `params` attribute.\n",
            "\n",
            "Please note that the `statsmodels` library in Python needs to be installed. If it's not installed, you can install it using pip:\n",
            "\n",
            "```\n",
            "pip install statsmodels\n",
            "```\n",
            "\n",
            "This code will generate a linear regression model where 'X1' and 'X2' are the independent variables and 'Y' is the dependent variable. The coefficients of the model will be printed to the console."
          ]
        }
      ]
    },
    {
      "cell_type": "markdown",
      "source": [
        "## Batch token streaming\n",
        "\"This method can improve the inference time.\""
      ],
      "metadata": {
        "id": "cvh_pS9SBE5t"
      }
    },
    {
      "cell_type": "code",
      "source": [
        "import torch\n",
        "from IPython.display import Markdown, display\n",
        "\n",
        "def hr(): display(Markdown('---'))\n",
        "def cprint(msg: str, color: str = \"blue\", **kwargs) -> None:\n",
        "    color_codes = {\n",
        "        \"blue\": \"\\033[34m\",\n",
        "        \"red\": \"\\033[31m\",\n",
        "        \"green\": \"\\033[32m\",\n",
        "        \"yellow\": \"\\033[33m\",\n",
        "        \"purple\": \"\\033[35m\",\n",
        "        \"cyan\": \"\\033[36m\",\n",
        "    }\n",
        "\n",
        "    if color not in color_codes:\n",
        "        raise ValueError(f\"Invalid info color: `{color}`\")\n",
        "\n",
        "    print(color_codes[color] + msg + \"\\033[0m\", **kwargs)\n",
        "\n",
        "cprint('Function to give color to texts', color=\"green\")"
      ],
      "metadata": {
        "colab": {
          "base_uri": "https://localhost:8080/"
        },
        "id": "-EKO4PdE0TU7",
        "outputId": "ed7569ad-e7ec-4d99-de21-bf8f23fd70f5"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[32mFunction to give color to texts\u001b[0m\n"
          ]
        }
      ]
    },
    {
      "cell_type": "code",
      "source": [
        "from transformers import AutoModelForCausalLM, AutoTokenizer, StoppingCriteria, StoppingCriteriaList\n",
        "from threading import Thread\n",
        "from IPython.display import clear_output\n",
        "\n",
        "model_inputs = tokenizer([prompt_template], return_tensors=\"pt\").to('cuda')\n",
        "\n",
        "# in the main thread. Adds timeout to the streamer to handle exceptions in the generation thread.\n",
        "streamer = TextIteratorStreamer(tokenizer, timeout=10., skip_prompt=True, skip_special_tokens=True)\n",
        "generate_kwargs = dict(\n",
        "    model_inputs,\n",
        "    streamer=streamer,\n",
        "    max_new_tokens=2000,\n",
        "    do_sample=True,\n",
        "    top_p=0.9,\n",
        "    temperature=float(0.7),\n",
        "    top_k=40\n",
        ")\n",
        "t = Thread(target=model.generate, kwargs=generate_kwargs)\n",
        "t.start()\n",
        "\n",
        "# Pull the generated text from the streamer, and update the model output.\n",
        "model_output = \"\"\n",
        "count= 0\n",
        "token_batch = 50\n",
        "\n",
        "for new_text in streamer:\n",
        "    model_output += new_text\n",
        "    count += 1\n",
        "    if count == token_batch:\n",
        "      count = 0\n",
        "\n",
        "      clear_output(wait=False)\n",
        "      cprint(prompt_template)\n",
        "      hr()\n",
        "      cprint(model_output, color=\"green\")\n",
        "\n",
        "clear_output(wait=False)\n",
        "cprint(prompt_template)\n",
        "hr()\n",
        "cprint(model_output, color=\"green\")"
      ],
      "metadata": {
        "id": "Hy7l8EatzEMh",
        "colab": {
          "base_uri": "https://localhost:8080/",
          "height": 1000
        },
        "outputId": "af38ca74-3d21-4bdf-9bb5-726e55caf603"
      },
      "execution_count": null,
      "outputs": [
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[34mBelow is an instruction that describes a task. Write a response that appropriately completes the request.\n",
            "\n",
            "### Instruction: Write a linear regresion in python\n",
            "\n",
            "### Response:\n",
            "\u001b[0m\n"
          ]
        },
        {
          "output_type": "display_data",
          "data": {
            "text/plain": [
              "<IPython.core.display.Markdown object>"
            ],
            "text/markdown": "---"
          },
          "metadata": {}
        },
        {
          "output_type": "stream",
          "name": "stdout",
          "text": [
            "\u001b[32m\n",
            "Here is an example of a simple linear regression in Python using the scikit-learn library. This code performs a simple linear regression on a dataset to predict the price of a house.\n",
            "\n",
            "```python\n",
            "from sklearn.linear_model import LinearRegression\n",
            "from sklearn.model_selection import train_test_split\n",
            "import pandas as pd\n",
            "import numpy as np\n",
            "\n",
            "# Load the dataset\n",
            "data = pd.read_csv('house_data.csv')\n",
            "\n",
            "# Split the data into training and testing sets\n",
            "X = data['Size']\n",
            "y = data['Price']\n",
            "X_train, X_test, y_train, y_test = train_test_split(X, y, test_size=0.2, random_state=42)\n",
            "\n",
            "# Create a linear regression model\n",
            "model = LinearRegression()\n",
            "\n",
            "# Train the model on the training data\n",
            "model.fit(X_train.values.reshape(-1, 1), y_train.values.reshape(-1, 1))\n",
            "\n",
            "# Make predictions on the test data\n",
            "y_pred = model.predict(X_test.values.reshape(-1, 1))\n",
            "\n",
            "# Calculate the mean squared error (MSE)\n",
            "mse = np.mean((y_pred - y_test)**2)\n",
            "\n",
            "# Print the coefficient and intercept of the model\n",
            "print(f'Coefficient: {model.coef_[0][0]}')\n",
            "print(f'Intercept: {model.intercept_[0]}')\n",
            "\n",
            "# Print the mean squared error\n",
            "print(f'Mean squared error: {mse}')\n",
            "```\n",
            "\n",
            "In this code:\n",
            "\n",
            "- We first import the necessary libraries: pandas for data manipulation, numpy for mathematical operations, scikit-learn for linear regression.\n",
            "\n",
            "- We then load the dataset into a pandas dataframe.\n",
            "\n",
            "- We split the dataset into training and testing sets using `train_test_split` function from scikit-learn library.\n",
            "\n",
            "- We create a linear regression model using `LinearRegression` class from scikit-learn library.\n",
            "\n",
            "- We train the model on the training data using the `fit` method.\n",
            "\n",
            "- We predict the values of y on the test data using the `predict` method.\n",
            "\n",
            "- We calculate the mean squared error (MSE) by calculating the mean of the squared differences between actual and predicted values.\n",
            "\n",
            "- We print the coefficient and intercept of the model.\n",
            "\n",
            "- Finally, we print the mean squared error.\n",
            "\n",
            "This code assumes that the input data is in the form of a pandas dataframe with 'Size' and 'Price' columns. If your data is in a different format, you'll need to adjust the code accordingly.\u001b[0m\n"
          ]
        }
      ]
    }
  ]
}